DOCUMENT RESUME 

TM 870 065 

Guthrie, John T. 

Roles of the National Assessment of Educational 
Progress in International Studies. 
23 Jul 86 

13p.; One of 46 papers commissioned by the Study 
Group on the National Assessment of Student 
Achievement and cited in Appendix B to their final 
report "The Nation's Report Card" (TM 870 049). For 
other papers in this group, see TM 870 050-094. 
Viewpoints (120) 

HFOl/PCOl Plus Postage. 

Achievement Tests; Agency Cooperation; ^Comparative 
Testing; Cross Cultural Studies; ^Educational 
Assessment; Educational Testing; Elementary Secondary 
Education; Evaluation Utilization; Foreign Countries; 
^International Cooperation; Measurement Objectives; 
National Surveys; ^Testing PrograKis; Test Use 
International Assn Evaluation Educ Achievement; 
^National Assessment of Educational Progress 



This paper discusses the rationale for international 
comparisons in educational achievement, presents factors which affect 
the validity of such comparisons, and makes recommendations for the 
role of the National Assessment of Educational Progress (NAEP) with 
respect to international studies. International comparisons are of 
interest in policy formation, resource allocation, and school 
improvement programs. Four prerequisites to international assessment 
also weigh heavily in an individual, national testing program: (1) 
the scope of item content must be equitable for each of the countries 
involved; (2) comparisons across countries will be facilitated by the 
use of common scaling techniques; (3) sampling must be representative 
and aedquate; and (4) the appropriate language must be used in 
testing. NAEP can facilitate international comparisons by 
collaborating with existing efforts such as those of the 
International Association for the Evaluation of Educational 
Achievement (lEA). NAEP can provide technical assistance in data 
collection and statistical analysis. An official liaison between NAEP 
and lEA is recommended at the policy level. An annual cooperative 
working meeting is also recommended. (GDC) 
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-BOLES CF THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS 
IN INTERNATIONAL STUDIES 
Purposes of Thits Paper 

The purposes of this paper are to: 1) discuss the 
rationaJ.e for irrtemational cxnparisons in education; 2) 
present factors that cif f ect the validity of such 
conparisans; ani, 3) make recammendations for the roles 
of NAEP with respect to intemationcd studies. 
H istory of International Comparisons 

Educational researdi in vMch the outcomes of 
different nations are coiroared has been conducted by a 
number of organizations incliodings" The Organization for 
Economic Cocperation and Development' (OECD) , The Council 
of Europe (OOE) , Tfae United Nations Educational^ Scientific 
ard Cultural Organization (UNESCO) and Ihe Intematioral 
Association for the Evalxiation of Educational Achievement 

(lEA) . 

The irost systematic and eitpirical of these 
organizations has been the lEA^ vdiich was begun by a groip 
of educations! researchers in the late 1950 's. Its. 
original parrpose was to ejqplore the effects of independent 



variables on student performance in several school subjects 
across languages arid across national toondaries. After a 
preliminary study of achieveirvent in matheiratics in 1966, 
lEA conducted a six-subject survey of performance in 
science, readir^, literature, civic educatiai,, arxi two 
larguages . (French, ard English) vMch was ccirpleted in 
1974. Fbllow-xp studies have: been conpleted in 
mathematics, and science. By 1980 studies of the classrocaa 
enviroonent ard written corrposition has been initiated. In 
laDst cases, the iiTterrjatianal reporl3 have presented 
general data about: the carparative jx^rf ormance of education 
systems and have ei^Dlained variations in level of 
performance in terms of variation in such areas as 
curriculum and instruction, school organization, ccmmunity 
and family, ai>d the personal characteristics of the 
students. Althjc-ut^ the ZEA dees not collect periodic 
data frcm a constant set of countries, it represents the 
itost well developed sou.'vce of international coitparisons 
involvirg USA ax>d other a.iuntries. 
Rationale for International Comparisons 

T^yarp at least three reasons for conducting 
educational research at an international level. First, 
measures of educational achievement vdiich provide 



cxmparisons of school outcomes are of intrinsic interest to 
a wide range of consumers. Researchers, policy-makers, 
teachers, and the public are continually curious about how 
children in the United States cottpare to children in other 
countries. As the United States becomes more involved with 
other nations on economic, and cultureil fronts, questions 
about conparative educational acccmplishments increase. 
Inlicato'rs of conparative progress in education attract 
growing public and political attention as nations becortve 
more i\^terdepenaent. 

A second reason for conducting international 
coiparasons in education is that policy-makers at the 
Federal, State and IxDcal levels value them for determinirg 
the allocation of resources. Ihe "standing of the US 
rslative to other (xxSitries in subjects such as reading, 
math and science, infl\ienca resources that are directed to 
these subjects. For exaitple, recent conparisons of the US 
with technologically advanced nations such as Japan and 
West Germani^ b^e contributed to the sense of need for 
i i i pr o v ement in science education. Policy documents such as 
"A Nation at Pis3c" have used international conparisons, 
derived from lEA data, to make 
the case that the United States is mediocre in its 



performance. Ihese data fueled the reform inovement of 
1983-1986. 

Ihe international data base on educational achievement 
is, regrettably, in a state of disrepair. A reporter from 
the Associated Press, vath whom I was taDcing today, was 
a^iast that the most recent data on reading was more than 
ten years old ard appalled that it included only 15 
countries. Since an international body such as the United 
Nations has not xinderta3cen systematic, periodic conparisons 
of education, equal in quality to indicators of our 
econamic health, it may be necessary for the USA throu^ 
NZffiP to exert leadership on this front. 

A tiiird reason for conducting conparisons of 
educational performance is the irtprovement of schools. 
Although NAEP has been devoted exclusively to data on 
school outcomes, data on iiputs are also valuable. 
Frequently, the approach to school iiprovement following 
international conparisons is haphazard. A country that has 
perfooBd poorly on an achievement measure will •underta3ce_ 
an inprovement program that is guided by fashion or 
ejqjedience rather than by data. In addition, educators 
frcm countries that have performed well on an achievement 



measure often insLke xmwa r ra ii ted cTaiTns about an aspect 
of the education system, such as the "teaching method" or 
the '^materials" vAaen those factors have not been the causal 
agent in producing acMevement. Althou<^ +"he data frm 
MffiP do not go beyond outcomes, the interpretations based 
on them nearly zdways extend to educational ii^xits. Such 
interpretations will be enhanced by information on the 
inputs of schooling. 

VJhen intemationcd carrparisons are used for school 
iirprovefflsnt^ data on educational inpaits as well as 
educational outcomes are valiaable. Typos of irputs that 
are useful incliode: instructional strategies, school 
resources^ student driaracteristics, educational materials, 
community context, and school policies. TEA has collected 
data on sorae of these variables in prior studies, but NAEP 
has not approached the issue of collecting data on 
indicators of educational inputs. Althou^ it may be 
premature to undertake national surveys of inputs since 
measureaaents of outcomes have not been perfected, it is 
sensible to forecast that these factors will inform 
school im p r o v ement efforts and it is reasonable to design 
assessments of outcomes that are cottipatible with school 
improvenent initiatives. 
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pequiregagrrts for International Comparisons 

Cciqparing the ecJucational achievement of students 
acrross cx3untries is a demanding exercise. There are at 
least four prerequisites to international assessment some 
of vftoidi edso weicfa heavily in an individual ^ national 
testing program. 

Scope , Ihe content of the assessnv?-nt items in an 
international stody must be equitable for each of the 
countries involved. At the heart of the issue .of scope is 
the opp or tu nity for students to learn the skills and 
Imowledje that are measured. Since it is loicwn that 
achieveae nt in sub-topics vdthin math and reading are 
associated vdth tinvs devoted to these topics in the 
curriculum^ the adequacy with vMch the assessrasnt 
represents the curriculum is an issue for many 
participating countries. Hie lEA has addressed this issue 
in the past by constructing an international core of tasks. 
All countries participating in a sbody are assessed on the 
core. In parallel^ a description of the opportunities to 
learn all diitvsnsicns of the core is cotpleted^ permitting 
b^'ai absolute (unwei^ted) and relative (wei^ted) 
corparisons across countries. 



Scales , corparisons across countries will be 
facilitated by the use of caramon scaling techniques. Not 
only are equitable tests needed, but car?parls>ons will 
benefit from scales that are constructed to describe the 
performnce of participants in a corainon metric. Ihe recent 
advances in scaling at the National Assessment of 
Bducaticnal Progress in the area of reading could be 
extended to an international data set. Ihe technical 
requiresnents of this extension are not trivial however, and 
working loeetings of content area specialists and 
psychometricians will be needed to address such issues as 
spiraliig^ wei^ted scoring, scale andioring, and 
inultidinensionality. Mbst of these problems can be 
resolved with existing statistical' techniques, alttiou^ 
their applications will require a diversity of e^q^ertise. 

Samplincf . Obtaining representative sairples across 
countries is a challenging logistical operation. Most 
countries have nationalized data hanks on schools, 
districts, pupils and personnel that permit stratified 
saitpling. The size of the sanple required for adequate 
representation varies with the intraclass correlation, 
which is the prcportion of variance in adiievement within a 
country that is attributable to differences between 
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schools. Countries in vAiicih differences between schools 
are large require larger sanples of students to keep 
sairpling error within tolerable limits. 

Executing the plan is a problem, Ctotaining a high 
rate of response to requests for participation in the study 
has been a more severe shortcoming in United States than 
other countries in prior lEA studies. NAEP could assist in 
addressing this issue. Consistently hi^ response rates 
across countries are necessary' to the creditability of 
international contrasts. 

Limuistic . T^sessroents across languages are 
inevitably difficult. In tests of math as well as reading 
and science, cross translation is needed to assure that the 
coherence of the taslcs and difficulty of vocabulary are 
similar in different languages. Furthermore^ tests aire 
\3sually administered in the language of the school^ but 
this is not necessarily the language of the student's home 
and the penalty .to bilii^gual populations must be addressed 
in the interpretation of findings. In imiltilingual or 
multicul'fcural settings the administration of tests is 
conplicated and special procedures aiB necessary to assure 
that students have undeirstood the tasks to be performed^ 
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and have attacihed appropriate inportance tjo the assessment 
activity. 

Facilitation of International Comparisons 

Ooiviparisons of educational achicvenvent that will 
eiiibrace the United States and other countries can be 
facilitated by the National Assessment of Educational 
Progress at l^ast three vays. Firsts NAEP can 'collaborate 
with international organizations tha;: axe collecting data 
on educational systems. The aost prominent of these 
organizations is the International Association for the 
Evaluation of Educational Achieverasnt (lEA) vfliich has been 
founded to foster such studies. 

The US participation in lEA studies in such areas as 
reading^ inath and science should be directed by committees 
of national e^^^erts in the respective content domains. The 
NZffiP could be contrac ted to provide services such as data 
collection, statistical analysis and consultation in 
accordance with tlie design of the national ani 
international committees that direct the studies. Thrcju^ 
this structure NAEP can lerd technical e^^jertise to the 
policy and research priorities shared by the USA and other 
countries in the XEA organization. 

Second, an official liciison between NAEP and lEA 
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should be established at the policy level. A member of the 
NMEP policy board .could work v/ith administrative officers 
of lEA to constnact concrete lines of noitual support. Ilie 
NAEP policy board could invite an officer of the TEA to 
serve as a voting or an ex-officio meiriber. A ineeting of 
minds art the policy lf2vel is necessary. 

Third, to foster coramuniciation, it is recommended that 
NAEP host a select, invited, vorkirg meetinj on an annual 
basis for exchange between NTffiP ani lEA research groi^js. 
OSie areas of reading, literacy, math and science could be 
represented at a mrilTnum. The design of assessments, 
sanpling schemes, item banks, spiraling or matrix sanpling 
procedures, data analysis, and archival strategies must be 
shared more broadly than they have in past. 

An exploratory meetir^ was held on May 6, 1986 vith 
NAEP reading and literacy staff, OERL staff, ai^ the lEA 
Literacy Steering Committee. After an exchange of general 
assessment designs and plans, the group concurred that 
joint data collection, analysis, reporting and publishing 
would be mutually beneficial. C3ontinuing the flew of . 
oommunicatian with the international researdi ccanmunity 
is a first step to enhancement of the role of NAEP in 
educational research on intematianal level. 
J!IG/md 



